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ABSTRACT 

This paper prbposes a strategy for in-house ^ 
evaluation? in the context of an educational research and development 
facility. The obstacles in conducting an evaluation of , colleagues • 
programs are- discussed; these fall into two categories. First, there 
is a set^of problems that relate to conflicts inherent in judging the 
work of a colleague without the protection of anonymity or 
organizational authority; and second, *there are problems posed by the 
unique characteristics of the program under consideration, which can 
appear more complex than they ate due to the^ myopia induced by such 
'close association. The proposed strategy, as presented, places the 
eValuator dn the role of coordinator^ facilitator, and synthesi?Qr of 
several separate research efforts in w^hich colleagues in a variety of 
disciplines combine en^rgie^ and interests to assess the program 
being examined. The paper states that the recruitment of researchers 
to participate in the strategy was accojnplished by persua:^ion and by 
assuring a payoff to the researchers involved in the form of 
contributions to their'^own area of research. The tactics used for 
organization, and management are discussed in some ^detail. The 
preliminary results suggested that the evaluation ejpfort was 
'facilitated and its value increased by multidisciplinary 
contributlo^ns to the ^esearcl^. (Author/RC) ^ 
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A STRATEGY FOR PROGRAM EVALUA'i;iON 

Gaea Leinhardt 

Learning Reoearcji and Development Center 
Univeroity of Ptttoburgh 



In thio paper I will deacribe a strategy for the in-houDc eValuatior^ of 
a specific curriculum in the context of an educational reoearch and develpp- 
ment center. The particular 'curriculum which ia examined ia the Individu- 
alieed Science (LS) curriculum (Champagne & Klopfer, 1974); the particular 
research and development center is the Learning Research and Develop- 
ment Center ( LKDC , Note 1). The need^for a new approach to evaluation 
in this context derives from the problems inherent in the role of an evalua- 
tor faced wtth making public judgments of the value of Colleagues' work 
and Ifrom those residing in the need to produce detailed and convincing evi- 
dence for an evaluation. In order to solve' these problems, a procedure 
\sL3 deVeIope(5 which places th^ evaluator in the role of coordinator, facili- 
tator/ and synthesizer of a. cooperative/^interdisciplinary research effort 
which brings a variety of research approaches to bear on a cluster of prob- 
lems. Here I will describe the fssues which led to the genesis of this pro- 
cedure, and detail the general strategy and tactics which were developed to 
improve its chances of success. Since the effort is on-going, there are no 
resuks in the usual sense. However, the unde rtaking "has led to the initia- 
tion of specific research projects whose results will bear on the overall 
evaluation, and these studies* are described. 

Problem 

The problems which confront a researcher engaged in evaluating the 

o 

- products of colleagues in the same organization fall into essentially two 
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categorieo. Firot, there io a oet of problemo that relate to conflicto in- 
herent in judging the work of a colleague without the protection of anonymity 
or organizational authot^ity (Scriven, 1,972). Second, there are pro*blemo 
pooed by the unique charaftter iotici3 of the pVogram \inder c onoide ration, 
which can appear more complex than they really are due to the myopia 
induced by ouch clooe aasociation. The strategy described in this paper * 
io deoigned to turn these apparent dioadvantages into advantages in the 
• evaluation bf educational programo. • . 

' As has been* docum'ented by Scriven (1972) and others (Thrasher, 

Franklin, Kittredge, Note 2),, it is often difficult for researchers in the 
oame organization to design and carry out studies whicji arq aimed at 
evaluating tKe work of their colleagues, especially when the results of 
^ tha^ research are to be.publicly consumable. Regardless of whether ouch 
research ip unbiased and objective, it will inevitably be viewed with suspi- 
cion by outsider^ if the results are hivorable. If the* results are unfavora-^ 
* V 

ble, a storm of prciteot will be raised about methodology, sensitivity, 
inoightfulneoo, and finally, the value of the evaluation. ^ Surviving such a 
otorm 10 difficult at a'diotance and debilitating when it takes place in one's 
own back yard. Inevitably, if there are positive resultsi the '''outside" 
world will be suspicious; if there are negative results, the "inside" world 
will be suspicious (see Rossi & Williams, 1972), 

... ' t 

Possible solutions to this dilemma have been debated in the literature. 
The most-popular solution*ds to remove the evaluative process from the 
scene o( develo^^frental effort either by bringing in evaluators who are 
external to the org anizfe1:ion or by placing fhe evaluators in a position of 



^See the now classic debate over the evaluation of Hetid irt tetween 
Cicirelli and Campbell and Erlebacher In Hellmuth (1970) anH a disci ooion 
of it by Williams and Evans in Rossi and Williams (1972). 
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higher organizational authority than 'the deVelopero (Thraoher et al. , 
Note 2). Unfortunately, thio oolution raioe^i more problemrf than it aolveo. 
"Outoi^e'ro" pay for their ability to be critical withauL^conoequence by a 
losti of knowledge and by a limitation on their sourcea of inf orma bion 
(Ro8«i, 1972), oAe way out of thiu -conundrum io to uae the potential for 
increaaed information, which the in-houjie rebearcher can have, to pro- 
vide Ihe moat convincing evidence for the value of the program. Hie trade- 
off i5i betwee-n*dubioufily credible objectivity and unique ac c e« oibility t>o 
informalion. Neutrality in no guarantee of high quality reaearch; neither 
doew {hf^*lack of neutrality inijure againnt it. In fact, efforts to preserve 
5iO- called neutrality can often lead to mediocrity in research by providing 
unbiased, "clean, ' but irrelevant data for evaluation, 

Evaluation can aim t(j be fierviceabl^ and ^/aive the ^normal guldelineo 
for rtMiearc h, or it can aini be reuearch and aeek to be aerviceable within 
that context. The in^ue ijj one of degree. Clearly an elegant reoearch 
report delivered three yearn af;ter the appropriate deadline ia of na uoe. 
On the other hand, a service document filled with theoretical or methodo- 
lot^ical er^rorH doen not truly fill the service function; instead, it merely , 
rloudH the iwsueH. I'he advantage of emphasizing the research aspect of 
evaluation is that it perrriitB the researcher to invoke the usual ruleo of 
evidence of rewearch. The evaluator can then assume a stance of support- 
ing or reje( tin^ the valtje of a particular product, in the same way a re- ' 
searcher favors a particular solution to a problem and supplies evidence to 
back up the position, ^ 

Kniphasizing either the re^^earch aspects or the service aspects of 
evaluation leaves untouched additional probletn's that arise from the fact 
that e^ch new program presents its ow^ set of problems for the refJearcher, 
For example, the I.RDC: science curriculum is extremely complex, haying 
multiple goals and multiple modes of accomplishing *tho8e goals. It is a 
program which not only focuses on inc reasing. cognitive abilities with respect 
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to Dcience, but which aluo attemptr; tij teach affective and attitudinal com- 
/ petencieo. Further, the oociety which <^*^amineH, pur^chasen^ an|| uBca 
the program doeo not have clear-cut j»oalb for elementary at^enoe educa- 
tion. If the, ^oalo a^re complex or if»^he conuilmersi have not atated with ' 
.clarity or connenouo, what they want fr9i1r^fin elementary science progra^i, 
it-io extremely difficult to provide evitienye that jdocumentH the value of 
the program, * » * 

Unfortunately, much of the work that han been done in the evaluation 
of educational programo by external evah;^tor« has ignored the iiinue of 
providing iioeful information ajid ha^^i instead focuBcd on a net of nide issuefj, 
Duch ao expe r imi?ntal ver^ua quaei- or Hon- experimental dejiigriH in educa- 
tion, Unleou ti*eatmentii are randomly aaaigned to the unit of analyaifj, 
truly comparable "control" groupu are almost never a^ilable^i'n educa- 
tional oettingD, Therefore, the enouing debate over the results of ab- 
called experimental otudieB tendn to ahift from an analysis of the program 
and itG effects to an argument over whether Ciroup A was ^ame aw, 
. equivalent to. or different from Cirpup B (Hellrnuth, I'^wO), This obaerva- 
tion ih not meant to imply a rejection of experimental defiign for eva-luation. 
It ifj rather an attempt to keep the real problem i,n racus and to suggest that 
valuable information in still obtainable t-hi-ough the application of different 

c 

techniquen. ^ • 

In summary, aever^al problems beset any effort to conduct evaluative 
reoearch in ediica.tion. The role of the evaluator can be a Bource of tension 
fof the evaluator and the organization. The fjjjecific program under investi- 
gation always contributes its own set of difficulties. Finally, problems of 
acceptable and realistic methodologies abound. In the remainder^of this 
paper I will examine one appro<{ch to solving some of these problems/ 
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* Before detailing- the otrategy, it io uoeful to deocrlbe briefly the 
oe^ing xind "the available reoourceo in v/hich the otrategy v/ao developed. 

The Learning R^^oearch and Dbvelopmen^Cente r io one of 14 feder- 
aHy funded educational laboratorieo and centero in the'United Stateo. LRDC 
io devoted to the improvement of education through'indiviJtla^izat^pn by 
adapting the content and pace of inotruction to the individual needo of the 
otudent (Glaoer, 1/^3). It panoioto of 23 research projecto, the majority 
of whic"h report 'directly to one of three C enter directorcr. Each project 
hao iTety/een one ar^ three Ph. I) level people and a oupport utaff. The 
Center hao attracted a number of profejjoional educatoro» poychologioto, 
and nociologiotD» all of whom work on diverge topic n which tend to con- 
verge in their purpooe if not in their approach. For the evaluation taok, 
a ouboet of that community hao be«'n draiyn together. The ouboet oonoiato 
of curriculum developero, evaluatoro,- and oome individuals leo tv dire ctly 
and leoo obvlouoly related to evaluation who are engaged ih baoic research 
in poychology, apthropology, and educational reoearch. 

The other reoourceo available for the evaluation effort conoiot of 
three local nchoolo which are uoing the science curriculum, Each ochool 
hao at leaf^t one science claooroom in which the Individualized Scieqce pro- 
gram io being uoed. Each science room oerviceo between five and eleven 
claooeo of science'. The three fjchoolo range in the degree to which the 
program is implemented and the degree to which the surrounding environ- 
ment oupporto individualized education. Two of the settingr ©^are develop- 
mental public ochoo^s^ with several individualized programs in operation, 
and the third is a parochial school which has no other formally individual- 
ized curricula. Although there i,s: no immediate access to "control" schools. 



^Developmental public schools are schopls with which LRDC has a cpn- 
tractual relationship permitting curriculum development work to be conducted. 

\ 
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, our experience hao been that limited amounto of reuehrch can be conc^ucted 
t\t other ochoolo in the Pittubufryh area, r.imited reuearch can be ca^rried 
out when a large -ocale effort involving teoting and other disruptive inter- 
vention would not be acceptable but omaller, uhort-term (one month) invee- 
tigationo would be. « 

i A Strategy • 

In order to effectively uue the reoourceo available in the Center to. 
provide evidence fx)r the value of the ncience program and to avoid ao many 
of the drawbacks ao ^oooible, the ochema represented in Figure;^ 1 (page ll) 
\ wao developed. The figure xliag rammatically illuotratt»« how opecific re- 
oourceo were brought to bear in solving the problem of eotabliohing the 
value of the Individualized Sciente curriculum, ^ The baoic strategy was 
to bring tagether the diveroe a^ilitieo and- inte re ot □ extant in the Center to 
inveotigate v^riouQ aopecto of the ^cience curriculum while oimultaneouoly 
offering reoearchero an opportunity to contribute in any way they wiohed to 
thoir own problem areao. Initially, th^ ochema was worked out with only 
vague inditationo of peripheral opecializationo. It wao not expected that all 
areao of the ocience program would be otudied, nor was it expected that 
each area would be otudied/rn equal depth. What wao exppcted was that the . 
collection of otudieo from differing peropectiveo would contribute greatly 
to the available knowledge about the way in which this j^^ecific program j 
worko.^ T>^ere io, of couroe, no guarantee thkt the IcnoVledge thuo generated 
\^iU be totally adequate to meet, the needo of educational conoumeru, but tra- 
ditional approaches to evaluation cannot guarantee that either. 



" Obviously, this entire effort depends upon both the cooperation and 
entJiuniasm of the entire Science. Projecjt. Without their generous support, 
none of the studies could have been conducted. 
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TacticD 

To operationalize thio otrategy, tactic o had to be develqiped to attract 
reoearchero from different groupo, to provide circumotanceo in which they 
could continue to work on their own areas while contributing to the aclence 
evaluation, and finally, to develop a setting in which ideas which were not 
fully formed could be aired and improved upon. The tactics used to imple- 
ment this strategy fall into roughly two c ate^jories : those behaviors asGO- 
ciated with initial selection and recruitment of researchers, and those 
behaviors asoocia-ted with development and maintenance of a task perform- 
ing g^^up. 

In theory, an optimal tselection of disciplines to be represented (i.e., 
the peripheral opecialtien) could be determined^a priori and people assigned 
to rdo opecific projects so that unique information would be maximized while 
overlap would be minimized. In reality, however^ while disciplines and 
research areas could be oelectecj in advance, people could not because 
1.RDC IS a oemi-hierarchical organization rather than a hierarchical one. 
Leadership is by perouaoion rather than by command. Therefore, research 
ers who were both available and in a relevant area had* to be attracted to the 
task and then worked with to develop studies which maximized the informa- 
tion each could contribute. ' 

Selection and recruitment . Initially, a list of individualjs who might 
be interested ih wo^rl^ng on the science evaluation effort was drawn up. 
Some individuals were included because they were connected with the 
-Science or Evaluation Projects. Others were added because their work 
or work interests indicated that their inclusion would be mutually benefi- 
cial. Still others were recommended or volunteered out of personal inter- 
eot as the initial meetings took place. Several graduate students werfc>. 
included on the list because It was assumed that the opportunity both to 
obtain thesis material and to develop pilot studies in a focused problem 
setting would be attractive to them. 
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After thia Hot vvao conntructed, indivldualu v/ere contacted one at a 
^.tirne and. the baoic probU;nrui were diocuooed. On the one. hand, the ocience 
prof»ram and the problem of dvaluatin^; it had to be made Intriguing, rele- 
vant, and Deductive. On the other hand, payoff to the i^ividuals involved 
In l«rrno of their ov/n baoic research interest had to be relatively anoured. 
In other vvordo, it v/ao important for the producth of reoearch to contribute 
to both the evaluation problem and the opeclfic dincipUneu^, ao repreoented 
by the double-headed arrown in Figure 1 (page 11). It wao felt that given 
acceoo to "natural" oettingo, many reoearchero would welcome the chance 
tocput oome of their worh to applied uoe. The reriKaining taok wao^to pro- 
vide a forum in which propooaln for reoearch and initial reoulto could be 
diocuosed in a oupportive v/ay. ' 

Maintenance , i^mall group meetings, v/hich includoct thpne reoearchi 
ero v/ho v/ere already "on board, " were held once every two weeko, vyhile 
the individual meetingn to recruit additional renearcherrt were being con- 
ducted. Several decioiono were made to increane the probability of the 
meeting.o' oucceno. They ^re to be open to interested reoearchero; how- 
ever-, an effort wao made to keep the proportion of active working partici- 
panto to oboervero high. Individualo were encouraged to attend meetingo 
or oend repr e oentative o to oee what they were about, but they were dio- 
couraged from continuing to attend if they did not intend to do any reoearch 
in the area. Ground ruleo for dine Tinoion of r(»oearch propoaalo were laid 
down .which encouraged directt-d, conntructiye c ritic inrrv and dl*ocouraged 
criticiom for ito own oake. 

The purpone of the meeting*] wafi to dincuBo one by one each of the 
propooed or in-proceon otudien. The difjCuoniorjn focuned on the relevance, 
feaoibility. and nignificance of each wt-udy. In riome caoeo the diocuooion 
pointed out no many problemn with a particular propooal that the Btudy was 
abarJ^ned. In other caoen diocufioions wandered away from the initial task 
of examining propooaln to excurHlono into what evaluation nhould or could be. 
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' Theoe wanderingo were not altogether purpoBeleHu for they helpefl to clarUy 
what the type -and extent of the reuultant in/ormatipn would be. Along with 
the general dincuooion. Bpecific methods for implementms each «tudy in 
field aettingu were oketched out. 

• There were drawbacka a li uoc iated with thu; proceuu. ' By having 
re,H-ar,hers from uo many areao all trying to work together, problemD 
<,r<.«e concemiing the^ individual rolen. 'l or example, uome reBearcheVo 
attempted to bee orjf^ e va luatp r n and entirely abandoned their own dioci- 
plinary perapectivL rather than merging the twq. Further, becauoe the 
n.ajority of the reueWiera Were neither evaluatora nor science educatoro, 
■problen;,. aro»e over thXeed to familiari/e everyone with science content, 
placing a burden on tile Scienc eVrojec t ntaff. Mnally, the moat ueriouo 
riak lay in the potential of producing diHparate fragn.ented atudiea which 
would have no common therpe or ia..ight. However, the majority of theoe 
. difficulties were overcome by continuouo diacuoiiion in the g roup meetingii 
during the firat two or three monthu Of initial work. 

Daring the firat Heven-n.onth period. 13 meetings were held. The 
firnt meetinga were largely organizational and introductory. In these meet- 
ing., the evaluatorn'functioned a« facilitator,,; they did not attempt to direct 
the kind of research to be conducted. Ah the meeting. progreBaed, a for- 
■ mat developed of preaentir,,- one or two propoaala for research or repotto 
of work in progrPaa per meetmg. Kach propoaal waa reviewed and dia- 
cuaaed with regard to both it . aubatance and ita feaaihility. It would be a 
legitimate critinhnTof thia approach that not enough initial guidance waa 
,,iven. ao-lhat discouraging fal^e starts were sometimes made. However, 
it waa felt that the error should he in the direction of too little rather than 
■ too much directiveneas. ar, this was consistent with the style of the organi- 



zation. 



F.ve montha after the m^^etinga began, an interesting inte r rupl^ion 
occurred. Research for Better Srhoola (RBS) requested a joint meeting 
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to exchange ideao on the evaluation of the science curriculum nince they 
were engaged in evaluating the IS field teot unito theoe became availa- 
"We. Their evaluation effort wau of the more traditionll outcome ixuHrua- 
ment type. ITie importance of the meeting fc^ uu wa.'i that it forced our 
group to oumma/ize exactly what we were doing^and why. Thin, in turn, 
led to a fnibfjtantial increaue in group fjolidarity. \ 

In uummary, recruitment wau by iifVitation .ind not coercion. No 
taoko or rolen wc»re ^noigned; they were nelec ti»d. Hie niaintenanc e of 
inferefit in the tank wao accomplii,hrd through highly ta -ik - o r ir nted and 
reanonably nuppo rtive g roup niee ting d. 

I Reiiultii to Dat«« 



Figure 1 ohownyhe ten banic wtudierj which currently m.ike up the 
group of ntu^H'o tc^obe done in science. The fipecific title of each «>tudy 
appearn in italic type on one nide of the double - a r rowed line; the primary 
diocipline from which the individual com'eu in in boldface type on the other 
oide. The area other than ncience evaluation to which the research con- 
tributes appears around the outnide periphery of the figure. I^e role of 
the evaluatoru is to help facilitate the research and, after all the ntudien 
are complete, to c oordinate renultn into a .ierieiJ of docun^entfi that can 

oerve evaluative functionn. 

♦ k 

Turning to the detailn of Figure 1, two of the evaluatore are Inter- 
eflted in validating tl^r model that had recently been proponed for inventi- 
gating classroom procennefj (Cooley ^ Lohnen, in prenn). One aspect of 
that modelVoncernn the use of time and itH effect on strident achievement. 
They investigated the re\ationship between the number of minuteB npent in 
science, math, and reading, and student achievement in one of the schooln 
in which the science program existed (Leinhardt R. Cooley, Note 3). Pre- 
liminary results indicate that fehe amount of time npent in acience, indepen- 
dent of entering abilities, in positively related not only to science, achieve - 

1 
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Figure 1. A strategy for science-curriculum evaluation. 
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thientr but also to math and reading achievement. This was supported by a 
sitnilar clinical finding previouoly reported by teachers and administrators 
using the^ocierxce curriculum. This finding has important implications *for 
the value of elementary science education to improve not only the standard 
• of scientific literacy, but also the overall levTel ol knowledge of elementary 
sch6ol children. A replication of this study is currently being conducted. 

V " O If* of the social psychologists is interested in studying teacher and 
Student perceptions in the classroom. The plannirfg for this research was 
completed before the meetings were started; however, the meetings wefe of 
. oome^help in implementing the plans. This researcher is planning k study 
in the science rooms to investigate student perceptions of learning, school, 
and thetnselves (Greenberg* Note 4). He is examining' the students' atti- 
tudes and beliefs about knowledge' acquisition and the roles of teachers and 
peers in different environments. This study is being conducted in an inner- 
city school and will provide information on how children view different learn- 
ing settings and will shed light on how those views might affect learning 
itseU. 

Another social psychologist is interested in the problems of locus of 
control and of changes in perceptions, which occur in different educational 
environments. He is conducting a study of locus of control within the con- 
texts of the science and mathematics clas s/oorpa at two grade levels in a 
parochial school. One section of mathematics is taught in a' traditional and 
nonindividualized manner, while science is taught in a more open and indi- 
vidualized way. He is working toward the description and measurement of 
locus of control which is situation specific rather than using the traditional 
view of It an unde rlying ^ait or characteristic 

The science educators, who have long been concerned with the prob- 
fems of formative^ evaluation and its documentation, are working with 
sociologists who are interested in documenting variations in the frames of 
reference of actors engaged in formative evaluation. The sociologists are 

12 ■ ' 
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conducting a case study of the-behavior s of the developers during a stage 
jt>f curricullum development.- The product of this joint effort will be a forma- 
tive evaluation of a specific: unit of the science curriculum and a detaUed 
description if how design decisions are made and by whom. Further infor- 
mation about\how the different groups involv^^with dissemination (pilblioh- 
ers and school personnel) impact curriculum development will also be docu- 
mented. * ' 

Two researchers are interested in developing methodologies for ana- ^ 
lyzing curricula. One of the two is interested in developing models of 
hierarchy analysis. She is' detailing the implied and overt hierarchies of 
both the affective/attitudinal and the scientific literacy goals of the curricu- 
lum The resultsW this will be a map of the science curriculum's objec- 
tives and the speciAic activities designed to meet these objectives. The 
second team of researchers is interested in developing measures of how 
closely cur'ricula us^ Icnown behavioral principles. These measures have 
been applied to several diverse curricula (Holland, 1975) and a detailed 
analysis of one of the Center's othejr curricula has already been conducted 
(Holland & Solomon, 1975). They are doing a similar analyat^s of the sci- 
ence curriculum. • 

One r^earcher is doing a small study of the science goal of student 
self -evaluation. She is interviewing students who have workeH through a 
• science unit before they take their posttest for the unit. In |he interview 
the students are asked to e^imate how much they know about a given sub- 
area. Th«ir estimation is then compared to theifyposttest performance. ^ 
In general, students tend to felightly underestimate their Icnowledge, with 
better stude-nts being closer in their estimates than poorer students. The 
importance of this study is that it will help to document the science activi- 
ties specifically designed to contribute to students' abilities to evaluate 
themselves, and it will also indicate how other curricula not so designed 
impede students' ability to evaluate their own knowledge states. 
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Two other atudiea have recently emerged in the science meetingo. 
One io an ethnographic fleocriptlon of atiident free-range behavlora in the 
ocienco room. The other ia the development of new and more extenaive 
science competency meaourea for elementary and aei;ondary achoola 
(labeled "Meaaurement" in ^^^oHflF U» Currently, moat of the standard- 
ized teajo do not distinguish oBfween acicitc^ anH social studies until the 
^Ifth grade. Further, the majority of existing tests are ap Imprecise and 
inaccurate in their language as to place a knowledgeable student at a dls • 
tinct disadvantage. , 

In summary, these studies contribute to the understanding of IS and 

0 

hence help to show the value of it along three dimensions. First, reasona- 
bly complete documentation of the content, pedagogical soundness, and 
design decisions of the program will be made available. Second, some 
Insight about specific classroom processes and their impact, both psycho- 
logical and social, will be generated. Third, the impact of science educa- 
tion on both science and other basic skill areas will be better understood. 

Conclusions 

An additional result of this exercise has beek the validation of the 
strateiiy presented in Figure I. Figure 2 represents a restatement of that 
str fifefry Independent of the specific studies being conducted. At the center 
of Figure 2 is the object of the evaluative inquiry. While the object of 
inquiry ia singular, the inquiries are considered plural. That is, there Is 
a set of problems or topics for Investigation. Around the peripliery of 
Figure 2 are several problem areas of research, the focus of which is not 
necessarily evaluative research. These peripheral foci are areas of 
research which grow out of specific discipline - related Inquiries. The figure 
Indicates that the problem areas and related disciplines can be, but are not 
necessarily, independent of one another. The strate gy is to incorporate the 
work of individuals from diverse backgrounds who are working on prjoblems 
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Figure 2. A strategy for program evalu^ion. 



15 A 



In their own 'diocipline areao into an overall program of evaluative research. 
Theoe individualo repreoent a powerful and generally untapj^ed reoource for' 
oolving many evaluative reoearch problemo. 

i The outcoi-neo: of bhio on-going effort are encouraging. If one wichea, 

one can exploit the available reoource^ pf a reoearch center to facilitate 
irt-houae evaluation in a creative faahion resulting In unique and uaable 
Information. The role of the evaluator i£3 tg orcheotrate the undertaking 
and, when t^e studies are completed, to incorporate them into an evalua- 
tiv© document. The product of ouch an effort can be an in-depth view of 
the program under investigation, one which is richer in expertise and Infor- . 
^^[lation thah any. single effort could hope to be. 

This strategy was designed to solve seve ral problems : the Conflict 
inherent in the role of In-houoe evaluator while providing convincing evi- 
dence for the value of a program and for the soundness of the evaluation; 
and the provision of more than 9upe rf ic iarl information about a highly com- 
plex program. The conflict of the role of the evaluator was fiolved in two 
ways: first, by sharing the burd]^ of evaluation among several res-e^ig^hers; 
and second, by including the developers in the group. Thin assured 
developers that misinformation and misinterpretation of informatjton at^m 
their program could be minimized. The problem of bias is not totally 
solved; however, it is reduced fey using a group of people whose primary 
-commitment is to good science in their respective disciplines. The problem 
of providing in-depth information about the program was solved^by having 
rrtultiple, and in some cases, overlapping studies conducted o^ varicrus facets 
of the program and by making use of already «>xisting re sour cea ,* many of 
which were previously untapped. 

An additional significance of this strategy for evaluation should n^pt be 
overlooked. One of the major complaints of social scientists (such as Rossi) 
is that the evaluation field has been unable to attract highly qualified indi- 
viduals and is the refuge of dropouts from other fields (Rossi &i ^yilliamo. 
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t The btrategy dencr^hed hero provider quality researcher b from 

other fields wijh an incentive to undertake work that Hvrvv\i <*valuative 
functions. Of course, evaluators must do research themselv<;s, but they 

• also muot continuouply eacourage their colleagtien who are not evaluatori* 
to engage in reoearch that can aloo serve the function of evaluation. Not 
only does this provide one with a oense of parsimony, it also educates and 
re^-educates evaluators by exposing the-m to the differing perspectives and 
methodoloj|ieB of their colleagues. A rerrmiViing question is whether or^ot 
there are other research organizations that h<lve sufficiently diverse person- 
nel to utilize this approach.' I think there ar«. For example, not only are 
there the unive r sity - related research centers, there also are governmental 
institutes and private industry laboratories, all of whicli have sufficiently 

r 

diverse professional staffs to permit utlization of this approach. 

^.The strategy Is not offered as a panacea for all the problems that 
face evaluation. It is offered as an alternative to the traditional in-house, 
one-shot studies conducted by a single team of evaluators. There is no 
iron-clad assurance that answers to a collection. of questions generated by 
another group\of researchers would provide adequate information a^out the 
value of a program. However, the strategy provides ample opportunit^for 
the evaluator and other rese^^rchers to discuss this point and try to con- 
struct a sefies of studies that will provide usable information ^f or consumers. 
The #m phasis in this strategy is for researchers to satisfy themselves with 
regard to the scientific excellence of their work and to seek answers to 
questions which they generate rather than attempting to second-guess the 
consumers' questions. 
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